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ABSTRACT . - 

Existing evaluation models have been examined arrd 
found wanting with respect to planning cooprehensive evaluation 
strategy. In seeking a m0r6 coherent conceptaal framework from which 
to design evaluation strategy, the Virginia Beach City Public Schools 
developed an "evaluation Systems Franjework. The framework represents a 
conceptual scheme which allows the lo.cal educational agency to 
delineate evaluation scope, define evaluator role's, and design 
evaluation procedures. (Author) 
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• " , INTRODUClljlOX 

\ 

The restiTucturing of public conf idence^and support figures' to be the 
^ most critical challenge facing our nation's schools during> the last quarter 

p-^ of the 20th century. Amid charges of inequality, inefficiency and irVele- 

t vance; accounta'bility emerged as an issue of national importance durxng 

the previous decade. More recently, congressional changes in the Elemen- 
tary and Secondary Education Act of 1965 have mandated tshe ayaluation of 
federally sponspred educational programs. Responding to these public 
' ' pressures and legislative demands, two significant factors have been ob- . 
'served v^ith respect to the field of evaluation. First, school divisions 

throughcmt the country have attempted to establish or expand evaluation ' . 
- capabilitiesr Second, .a proliferation of evaluation models have been gen- 
• ' * erated .(Wortheh, 1968; Anderson, 1973; Carter, 1975). Yet, questions arise 
as to whether excessive demands have far exceeded theoretical developments 
in the field. In an attempt. to address these issues from a practitioners 
perspective, a structural fraraev/ork has been suggested to aid the Local 
' Ediicational Agency in 'the^conceptualizatlon of evaluation strategies. 

• • > 

' , THE NEED FOR A c'ONCEPTUAl/ FRAMEWORK 

CO ' , ' 

One of the principle prcblras confronting the school division V7hich 
' plans to develop or expand its evaluation capabilities, is the lack of a 
» cpncoptua^ framework from which to def ino. evaluation funct^n and evaluator 
' roles* Often the c^aluat ;on .theorist serves only to confuse rather than 

ERIC , . - . 



' ' ' • ConccptJializirig.r.vaU.ati'on S-crategy; 
An Evaluation Sys tenia Framework' 

J 



clarify >evalua tor roLcs in bis efforts .to- define the parameters of this 
newly emerging field* Evaluation, we 'are told, by natui;e involves the 
* formulation of judgments concernftig th^ worth o£ educational pheliomena,^ ^ 
and theiiefcre, the"evaluator> principle function' is that of valuing 
(Popham, 1975). \vhile .few would dispute the fact that .evaluation necessi- 
' tates the- weighing of m^rit, it may prove presumptuous to assume that 
such responsibility rest.s solely ul'th the evaluation practitioner. The 
eyaluator functioning within the LEA often finds^ thet sucfi decisions are 
reserved for the admin j stra.tive hierarchy,. And, at this level, subst&n- 
tive evaluation (i.e., empirical evidence) dnly'-con'stitutes one , criterion 
uporf whi?:t\, judgments of worth are made. ^i$s 'House (1973) suggests 
factors such as timeliness, pervasiveness, political' pressure, and public 
' opinion surface to influence the ^decision process. 

The evaluator, as Airasian (1974) observes, can exert -little control 
over the motives or attitudes of his superiors; nor^ can* he dictate to 
what extent his da^ta will utlimately dnf luence the decision-making process 
All he can attempt is tp provide the least ambiguous and most accurate 
information regarding decision alternatives.^ However, decision-makers/ 
must delineate the sdope and function ^of the evaluation effort prior 
to the' identification of evaluator roles and the securing of relevant 
'data. It is in this .tegarjd that we have labored to def inc^ ^n Evaluation 

Syst.pms Framework (ESi*')-; , 

Models designed to eicpedite the- practice of evaluation abound. How- ^ 
eve#, while varying to a degree In approach, most models appear markedly 
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alike with respect to their prcnounc^cl preoccupation with the scrutiniz- 
ing of prbgram process. While process plays a significant role with 
respect to systems thepry, the decided emphasis placed on program opera- 
tion leads one to suspect the political underpipings of many current eval- 
uation models,' Existing evaluation models-^may prove useful for the moni- 
"toring of educational programs; but contribute little to the overall 
'conceptualization of a comprehensive evaluation strategy, with the possible 
exception of Stuff lebeam's Context-Input-Process-Product Model (Stuffle- 
beam, 1971) or Alkin's CSE Model (Alkin, 1969; Alkin, 1974). 

Productive Evaluation at the local level rests with ability of - 
the LEA to effect the following items: delineate -evaluation scope; define 

V 

\ 

evaluator roles; and describe evaluation functions. 'Inmost instances, 
it proves counterproductive to enlist the services of a model without" 
first having the benefit of a well-defined structural framework. 'The 
failure o'f existing models to deal adequately with these and other related 
structural questions contrib,utes to the restrictive nature of current 
evaluation theory ; , Again from the practitioners perspective, program 
evaluation must 'seek the confeinuity^of^a^ total "assessment (i.e., from in- 
caption to termination), and attempt to rei^te findings tp ,the larger 
social and educational context* in or3er.^ be of value to the educational. 
. decision^ maker , Therefore, it behooves- those of us in the field to seek 



new conceptual framework^ from which to prbcped 
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TOWARDS A TOTAL EVALUATION STRATEGY 

Seeking to provide a more comprehensive conceptual framework from 
which to design evaluation strategy, the Virginia Beach City Public 
School^ has developed the ESF. In an effort to dispel any misconception 
at the onset, it may prove useful to delineate precisely what the ESF 
and what it is_ not . It is not an evaluation model, for it was designed to 
play a more structural rather than technical role (i.e., a system fj? 
genearating strategy) . The ESF is, howeVer, a conceptual scheme whiph 
allows the local educational agency to define evaluation scope, identify 
evaluation staff, and delsign evaluation procedures. 

For our purposes, the LEA has been defined as a system functioning 
within socio-cultniral/socio-political environment (see Figure I) . The 
system, as Easton (1965) has suggested, continually experiences ^'demand 
stress** which may be generated from internal or external sources. Once 
these demands gain recognition within the system, programs are usually 
designed to address specific needs. These programs^ characteristically 
progress through tl\ree phases, formation, implementation and termination, 

which coincidently parallel the inpat-process-output systems format. 

1 

Associated with each of these thr^e pragmatic 'phases are Certain evalua- 
tion function^ and related evaluafcor roles which mast be identified prior 
to t.he planning of evaluation stifategy. Despite current practice, attempts 
to assess ,the total impact of a program upon tH^i system must involve an 
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examination of all three phases < 



FIGURE I 

Tl\e Life Cycle of an Educatdonal 
Program: A Conceptual Model 
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Managing Stress 

The first and perhaps most legitimate concern with respect to planning 
evaluation strategy, involves an assessment of stress factors which give rise 
to new educational programs or practices (see Figure I) . The importance of 
this stage of the evaluation is underscored by two primary factors. First, 
the program at this phase is most directly associated with the underlying 
purpose of the system. Second, developments at this phase influence the form 
^nd delivery of the program in subsequent phases. Thus, the management of 
stress becomes the focus of the first component in our framework. 

The evaluator's role in this formative phase is that of program coor- 
dinator, ^or he must ser^Te as liaison between program developers and cvalua- 
tors (see Figure II). The evaluatojc, at this juncture, must g(^nerate infor- 
mation which will aid in the analysis of stress factors and their related 
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need xequirferaen ts . The data collected w^-thin this phase should contribute ^ 
to a broadet understanding of program soals, while providing a basis for , 

planning future evaluation 'strategy . / , . * ' ' 

' . ' • ' - ' ' ' . * ; ; 

Characteristically, this component* or phase of the evaluation- involves 

the following items: ' - * 

/ • 

1) gaining a familiarity wi,th the phenomenon In question- 

2) providing a census of problems regarded as- relevant 

by people working in the -area of concern. , 

3) clarifying concepts and relationships between variables. - , 

4) establishing criteria for future evaluation efforts. 

• . * . * 

The major concern at this stage ^of the ^valxiation is tjhe clarification 
of stress factors for goal setting purposes. As the initially vaguely . 
'defined problem becomes transformed into one with more precise "ineaniiig, ^ 
numerous adjustments may be needed with respect* to the formulation of pro- 
gram goals and evaluation design. < 

The evaluator's o^fLcntation during this initial phase must be one of 
alert receptivity, of seeking ratiier than testing. He can anticipate that 
his design will be subject, to frequent >-evision as new information is generated. 
Changes may become necessary in both the amount and type of data required. 

Data gathering procedures may include unobtrusive methods such as 
participant observatio^i, documents research^ and anecdotal records (Brandt, 
1972; Webb et al,,'I966), as well as, more reactive techniques such as inter- 
.views and questionnaires. Again^. these procedures are, subject to vary as 
the problem becomes more clearly defined. 
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Monitorin g Progr ess '* - 

The second element ot the framework focuses on the monitoring of 
pifogram progress (see Figure II), Assessment at this.' phase of program pp- 
eration has been variously termed formative or process evaluation. Eval- 
uation at this level m^y or may^ not serve as a basis for program modifica- 
tion-, ^depending, upon the requirements placed on administrative cJ^cisions. 
However, evaluation strategy at this stage must concern itself with the 
re-examination and clarification of program objectives. Therefore, whether 
or ftot program monitoring serves a developmental function, it most always 
serves *a descriptive function (i.e., supplying information concerning pro- 
gram implementation) . . . * , , » . 

The evaluator^'s role is now defined as 'that of program manager., The 



program Ihahager-* s principfdk purpose is to ensure that . discrepancy between 
initial intent and current ^operations does not impede program eff ect:|.veness. 
He must be in a position to deal with the following questions. 

I) ;llas program .philosophy changed? 
' 2\ Haye program objectives shifted? ^ • 

3) Need staff roles be redefined? ' ^ ^ 

^ ■ ^ Research procedures ^j^propriate for the evaluation at this phase in- 
clude such techniques as logical* an^lysis^ content analysis, and critical 
incident. Both' reactive and honreactive ^measures may^ be employed for the 

purposes of data ^^atherihg.^ In addition, any one of. several evaluation mode 

^ y ■ 

(e.g,, Tripodi's Differential Evaluation Model\ of PtoVus '.Discrepancy Eval- 
uation Model) may be applied to lend added structure ,t:o the asses^sment of 
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-program progress, 



Measuring Product 

— - — .- w ■ — ■ c 

The final component of the evaluation framework necessitates the > 
quantitative analysis of program products. The significance of this phase 
of the evaluation lies with the grounding of administrative judgment in 
empirical evidence. Summative or ptoduct evaluation provides the decision- 
maker with the hard data necessary for assessing overall program effective- 

ness. . 

Product evaluation involves the formal assessment of program cri- 
terion or output. Generally, program outcomes are measured and assessed 
through the. use of a comparison group(s)'. In cases^ where multiple or . con- 
• trol group designs are appropriate and feasible the evaluatcr may wish to 
consult Campbell'and Stanley's (196?) classical treatment of experimental- 
design or Pcpham's (197?) ^suggested adaptatjions f-ar the field of evaluation. 

However, in oth^er cases, desi'gns employing mul-tiple groups may prove 
i-nappropriate, and/or impractical. Under these circumstances it m^x he 
desirable to compare program outcomes ag^iinst an arbitrary standard, defined 
earlier during the initial phase- of program d,evelopment Horst, et al. (1975) 
suggest the use of a •»Nori:i-ref crenced^^vaXuation design when comparison 

'A ' / ' ^ * ' " ^ 

groups are u.navailable » '^ ' • • 

Regardless whether .pr not obntror- group designs are "employed; the . ^ 

primary, function of this -phase of thP, evaluation remains that of providing ' 
■ ■ ^ . - ■■ , . ■. 

^ the dccision-mak^r wi th 'empirical evidence concerning the realization of. 
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program goals. Due to these information rvequirements the evaluator must 
possess research competencies in the areas of expeririien tal design an^ 
quantitative analyses (see Figure II) • 

» 

Although the importance of quantitative assessment has been under- 
scored in this section, desired program outcomes are often of such a^com- , 
plex nature that they cannot be properly measured or inferred by means of 
a singled crit^itrn, as Moursund (1973) has noted. In this event, a broader 
data base may be necessary in order to accurately describe program benefits. 
Informal interviews and observations, pro j" act records and unsolicited comments' 
may provide alternate sources of information concerning the effects of pro- 
grani outcomes (Zelditch, 1972). , 
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, ' A FINAL WORD . * 

\Wat we have attempted to'provide in tlie above discussion is. a con- 
ceptual framework within which evaluation scope and , fUnctfon, including^, 
evalua'tir roles, can be identified.' We hope that through the use of '^uch a 
framework administrators., as well as evaluators, will be in a better posi-. 
Lion to plan more comprehensive evaluations. Evaluations which will yield 
more meaningful and useful inf ormation/or channeling into the decision ' > 

process. ', ■ , 

two ^observations should perjiaps be noted in closing. First, the ^ 
"logical ordering of the framework's components ,(i-e., .paralleling pro- 
" grammatic phases), Ooes not necessarily imply'a temporal ordpring." Indeed, 

design and analysis considerations, have been known to surface early dur- 
• ing<the fomiulative - stages of ptogram planning. And second, any one^ or 
all of .the evaluator- roles , described above may be occupied by one or more 
individuals. •' ^ • . , 
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